Evaluation Theory for 
Developmental Mathematics 
Practitioners 

This article is designed to present an overview 
of critical theory, research, and evaluation 
for the developmental mathematics educator. 
Students caught in the gap between high school 
mathematics preparation and entry-level college 
mathematics expectations — developmental 
mathematics education students—need to have 
Irene Mary Duranczyk their personal narratives told and have the 

University of Minnesota measures of success reflect their needs. Highlights 

of evaluation theory and research, as well as a 
combination of qualitative and quantitative 
methods are presented for the developmental 
mathematics classroom researcher. There is 
an ongoing need for practical information 
on the effectiveness of programs and services 
addressing the mathematics educational gap. 
Hopefully this article will help you define your 
role in advancing this important evaluation 
and research area. 

Whenever college populations are expanding and financial resources 
are diminishing, the role of developmental education is questioned 
(Saxon & Boylan, 1999). Many studies have demonstrated the 
effectiveness of developmental education through a variety of research 
methods (Boylan, Bliss, & Bonham, 1997; Boylan, Bonham, Claxton, 
& Bliss, 1992; Roueche & Roueche, 1993; Thomas & Higbee, 1996; 
Waycaster, 2001), especially in terms of student retention (Durant, 
1992; Lyons, 1990; Simmons, 1994; Umoh, Eddy, & Spaulding, 
1994). Politically, this research has not been reason enough for higher 
education to embrace developmental education, particularly at 4-year 
colleges and research universities (Jenkins & Boswell, 2002). This 
article will outline the history and benefits of three research areas (out¬ 
comes theory, retention theory, and attribution theory) that can assist 
developmental mathematics education practitioners in determining 
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what data will assist in program evaluation and will capture the 
benefits of developmental mathematics education for individuals 
who have completed their secondary education successfully and are 
deemed underprepared for postsecondary educational services. 

Outcomes Theory and Research 

In each decade since 1960, at least one outstanding outcomes 
theorist in the evolution of evaluation theory stands out (Fink, 1995; 
Lancy, 1993; Shadish, Cook, & Leviton, 1991). Social program 
evaluation theories emphasized a search for truth about effective 
solutions to social problems in the 1960s. Although Scriven (1983) 
and Campbell (1963) exemplify this period, the controversial work 
of Scriven on unanticipated outcomes or needs-referenced evaluation 
particularly highlights capturing the student’s voice. “Programs, like 
products, should be evaluated by matching their effects against the 
needs of those whom they affect” (Scriven, p. 235). Scriven’s model 
for evaluation called for an outside evaluator who would not reference 
program goals prior to data collection—who would evaluate the 
program solely based on participant perceptions—without prior 
institutional or political program knowledge. 

In the 1970s, Stake added the concerns of how social science 
concepts and findings could produce political and socially useful 
results (Shadish et ah, 1991). In this vein, Stake advocated qualitative 
methods for social program evaluation and emphasized the role of 
the case study in evaluation to find useful input-output relationships. 
Stake, like Scriven (1983), did not emphasize management concerns. 
Underlying Stake’s approach, however, is “concern for stakeholder 
well-being, identification of the particular stakes that persons have in 
a program and a desire to serve those whom the program is supposed 
to be helping” (Shadish et ah, p. 273). 

In the 1980s, Cronbach and Rossi (Payne, 1994) stand out. Both 
placed greater emphasis on evaluation to facilitate the transfer of 
knowledge for social program improvement throughout the U.S. The 
voices of all stakeholders are of equal value. Cronbach advocated the 
use of quantitative social research, history, ethnography, journalism, 
and critical reflection as evaluative research methods. 
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Outcomes theory and research was used primarily to evaluate 
social programs, not educational programs. Quantitative research 
dominated educational research. Numerous quantitative studies have 
indicated that educational and social programs did not have significant 
outcomes. In response to this trend, Patton (1990) advocated the use 
of case studies: 

Case studies are manageable, and it is more desirable to have a 
few carefully done case studies with results one can trust than to 
aim for large, probabilistic samples with results that are dubious 
because of the multitude of technical, logistic, and management 
problems, (p. 100) 

Patton said that most program evaluation is based on the false 
premise that educational interventions are true experiments, when, in 
fact, uneven implementation of programs, self-interest of participants, 
and difficulty of specifying—let alone measuring—outcomes makes 
it too easy and too likely to explain away or ignore negative results. 
Qualitative research, relatively open ended and concerned with how as 
well as how well, can much more honestly depict contextual factors. 

Through the work of Freire and Faundez (1989), Stage (1992), 
and Stage, Muller, Kinzie, and Simmons (1998), research in higher 
education has moved into critical research. Stage (1992) urged 
researcher [s] to move beyond explanation of what is happening 
today and focus research toward attempts to influence future 
possibilities. This perspective, critical theory, may be helpful to 
those seeking new ways to gather data on their campuses to effect 
change in those environments, (p. 2) 

This philosophy has provided the foundation for current practice in 
evaluation in higher education. 

Retention Theory and Research 

One area of evaluation that continues to be a high priority for 
postsecondary institutions is the study of student retention. Tinto 
developed an interaction theory model that contends that retention or 
attrition results from the holistic interactions between a student and 
the collegiate environment and not solely from individual attributes, 
program components, or the environment. Tinto’s model has driven 
much of the retention research and has been used by a number of 
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developmental education researchers to measure student persistence 
(Nora, Attinasi, & Matonak, 1990). Nora et al. contended, “Student 
persistence studies have not found that precollege factors have a 
significant impact on retention, but this lack of extensive direct effects 
may result from misspecification errors because studies have not 
incorporated appropriate indicators in quantitative models” (p. 338). 
The researchers hypothesized that getting ready for college would 
be a significant precollege indicator of persistence. This quantitative 
study, however, revealed that getting ready for college actually had a 
negative direct effect on retention. 

Reflecting on Patton’s (1990) evaluation of narrow-focused, 
quantitative studies suggests that a more exploratory study, using 
qualitative methods, might discover why students persist, even if 
they are underprepared, have histories of failure, or are identified as 
developmental education students. Would qualitative research on 
nontraditional students also indicate that precollege getting-ready 
experiences have a negative impact on retention? “There is only so 
much of human behavior that can ultimately be captured in numbers. 
The researcher needs to ground his or her understanding of what 
happens to students in college in the students’ own understanding of 
these events” (Attinasi & Nora, 1992, p. 25). 

Tinto’s (1993) model has also been used to predict success. A 
critical ethnographic study seeking to understand why and how 
developmental education students persist may provide more insight 
into underprepared college student retention. Certainly, factors 
of adult motivation may be significant in predicting or promoting 
the success of nontraditional students. Studies by Umoh, Eddy, and 
Spaulding (1994) and Whiteley and Fenske (1990) added further 
uncertainty to the topic when they used the Tinto model with post¬ 
secondary developmental mathematics students. 

Whiteley and Fenske (1990) conducted a longitudinal study 
examining ways in which college mathematics influences stability 
and changes in students’ final choice of undergraduate majors. The 
independent variables were mathematics exposure, high school grade 
point average (GPA), college GPA, gender, race and ethnicity, ACT 
mathematics score, college mathematics experience, non-mathematics 
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academic experience, and shift to final major. Whiteley and Fenske 
concluded that a complex relationship existed: 

The interaction of the college mathematics experience with both 
the student and the institution are complex and should not be 
oversimplified across college majors and/or student preparation 
levels.... Our findings strongly suggest that it is time for researchers 
and policy makers to move away from the simplistic and now out¬ 
dated notion of mathematics as the “critical filter” and toward a 
new focus on research and thinking about college mathematics. 

(p. 382) 

Umoh et al. (1994) examined the relationship between retention 
of students in 2-year developmental mathematics programs and 
several variables: age, gender, parents’ education, GPA, academic 
goal commitment, academic integration, institutional experience, 
placement test scores, and student performance. They found no 
statistically significant relationships between the independent variables 
and retention of developmental mathematics students. However, they 
did make the following comment: 

Developmental education students differ from typical college 
or university students because neither grade point average nor 
academic achievements are factors in determining retention 
in developmental education mathematics programs. Students 
taking developmental education mathematics are not forced out. 
Retention in developmental education mathematics seems, there¬ 
fore, to be based on an individual student’s intent to continue his 
or her studies, irrespective of getting good grades, (p. 42) 

Attribution Theory and Research 

Another important area of research related to understanding student 
trends in developmental mathematics, yet quite limited in scope, is 
attribution theory research. This body of research has focused on the 
Adult Mathematics Attribution Scale (AMAS) by Lehmann (1987) 
and the Mathematics Attribution Scale (MAS) developed earlier 
by Fennema, Wolleat, and Pedro (1979). These instruments isolate 
attribution of success and failure to ability, task difficulty, effort, or 
luck. The study by Lehmann included nontraditional students and 
found neither significant difference in characteristics (attribution of 
success and failure or pre-post course measures) of traditional and 
nontraditional college freshmen taking a developmental mathematics 



6 


Evaluation Theory for Mathematics 


class nor a significant correlation between attribution and pre- or 
post-course measures. 

Bempechat, Nakkula, and Wu (1996) used attribution theory as 
a predictor of mathematics achievement. They asked the question, 
“Do high and low achievers differ in their attribution patterns, and 
if so how?” (p. 54). Studying sixth graders, they concluded that high 
achievement was associated with attributing success to ability. There 
were a few more studies in the last 20 years attempting to relate 
attribution and achievement in mathematics students at the college 
level using the MAS and AMAS scales. 

Dependent Variables Informing Educational Research 

Penny and White (1998) conducted an ex post facto multiple 
regression analysis of selective characteristics of developmental 
faculty (gender, age, educational preparation, teaching experience, 
and employment status) and students (gender, ethnicity, age, and 
enrollment status) to determine which attributes are significantly 
related to student performance in developmental mathematics and 
their subsequent college-level algebra course. Their study revealed that 
students’ performance in the last developmental mathematics course 
was the strongest predictor of their performance in college algebra. 
The study also revealed that part-time enrollment and traditional 
college age had a negative effect on student performance. The impact 
of age on performance was not supported in the research conducted 
by Burgess (1992). In that study, younger students performed better 
in all levels of mathematics than nontraditional age students. 

These studies, along with quantitative studies conducted by Durant 
(1992), England (1993), Feingold (1994), Lyons (1990), Seybert 
and Soltz (1992), and Short (1996), identified characteristics of 
developmental education students that affected their performance or 
predicted success. The question that remains unanswered is: Why do 
these characteristics have an impact on success? A qualitative study can 
shed light on the findings of quantitative studies. Qualitative studies 
provide the thick descriptions needed to understand the connections 
between student characteristics and performance. Qualitative studies 
can reveal and discover other characteristics that may be significant 
or underlie the characteristics revealed in quantitative studies (Eisner 
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& Peshkin, 1990). The impact of developmental education is a 
complex field of study which will not be understood using any single 
methodological approach. This article does not have answers but raises 
qualitative questions for the practitioner to ponder and then proceed 
to design a research study which will “render tone, tint, texture, and 
nuance” to quantitative approaches which provide only the “broad 
outlines of the portrait” (Pascarella & Terenzini, 2006). 

Conclusion 

There is much more evaluation work that can and must be done to 
make the case for developmental mathematics education and its benefits 
to society, institutions of higher education, and all postsecondary 
students. American College Testing (ACT, 2005) reported that only 
41 percent of the high school graduates who took that ACT in 2005 
scored a 22 or higher on the ACT Math Test, indicating they had 
a high probability of succeeding in college algebra. That leaves 59 
percent of the high school graduates in 2005 demonstrating less than 
college level skills on the ACT Math Test and possible candidates 
for developmental mathematics course work. The number of students 
underprepared in mathematics seeking postsecondary options will 
continue. How will we address this need? How will we present the 
effectiveness of our developmental mathematics education components 
and programs? Hopefully the research methods presented in this 
article will help you think about new approaches and possibilities for 
evaluating developmental mathematics programs. 
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